{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "23777228",
   "metadata": {},
   "source": [
    "## 多线程爬虫"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "975b85f2",
   "metadata": {},
   "source": [
    "我们之前写的爬虫都是单个线程的，一旦一个地方卡住不动了，那不就永远等待下去了？为此我们可以使用多线程或者多进程来处理"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f781ac03",
   "metadata": {},
   "source": [
    "爬虫使用多线程来处理网络请求，使用线程来处理URL队列中的url，然后将url返回的结果保存在另一个队列中，其它线程在读取这个队列中的数据，然后写到文件中去"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "afac729b",
   "metadata": {},
   "source": [
    "### URL队列和结果队列\n",
    "将要爬取的 url 放在一个队列中，这里使用标准库Queue。访问url后的结果保存在结果队列中\n",
    "\n",
    "初始化一个URL队列："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6bab93d1",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-25T01:53:02.653170Z",
     "start_time": "2023-10-25T01:53:02.642540Z"
    }
   },
   "outputs": [],
   "source": [
    "from queue import Queue\n",
    "\n",
    "urls_queue = Queue()\n",
    "out_queue = Queue()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "c27cb7c0",
   "metadata": {},
   "source": [
    "**Queue模块中的常用方法:**\n",
    "\n",
    "Python的 Queue模块 中提供了同步的、线程安全的队列类，包括FIFO（先入先出)队列Queue，LIFO（后入先出）队列LifoQueue，和优先级队列PriorityQueue。这些队列都实现了锁原语，能够在多线程中直接使用。可以使用队列来实现线程间的同步\n",
    "\n",
    "- Queue.qsize() \n",
    "    - 返回队列的大小\n",
    "- Queue.empty() \n",
    "    - 如果队列为空，返回True, 反之False\n",
    "- Queue.full() \n",
    "    - 如果队列满了，返回True, 反之False\n",
    "- Queue.full \n",
    "    - 与 maxsize 大小对应\n",
    "- Queue.get([block[, timeout]]) \n",
    "    - 获取队列，timeout等待时间\n",
    "- Queue.get_nowait()  \n",
    "    - 相当 Queue.get(False)\n",
    "- Queue.put(item)  \n",
    "    - 写入队列，timeout等待时间\n",
    "- Queue.put_nowait(item)  \n",
    "    - 相当Queue.put(item, False)\n",
    "- Queue.task_done() \n",
    "    - 在完成一项工作之后，Queue.task_done()函数向任务已经完成的队列发送一个信号\n",
    "- Queue.join()  \n",
    "    - 实际上意味着等到队列为空，再执行别的操作"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "8837d97b",
   "metadata": {},
   "source": [
    "### 类包装\n",
    "\n",
    "使用多个线程，不停的取URL队列中的url，并进行处理："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "9aae0358",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-25T02:01:15.814084Z",
     "start_time": "2023-10-25T02:01:15.805802Z"
    }
   },
   "outputs": [],
   "source": [
    "import threading\n",
    "\n",
    "class ThreadCrawl(threading.Thread):\n",
    "    \"\"\"爬取数据的线程\"\"\"\n",
    "    \n",
    "    def __init__(self, queue, out_queue):\n",
    "        \n",
    "        # 线程初始化\n",
    "        threading.Thread.__init__(self)\n",
    "        \n",
    "        # URL队列\n",
    "        self.queue = queue\n",
    "        self.out_queue = out_queue\n",
    "\n",
    "    def run(self):\n",
    "        \n",
    "        # 取出 URL队列\n",
    "        while True:\n",
    "            item = self.queue.get()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b82f3c50",
   "metadata": {},
   "source": [
    "如果队列为空，线程就会被阻塞，直到队列不为空。处理队列中的一条数据后，就需要通知队列已经处理完该条数据"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "f1b10c9f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T00:13:21.193924Z",
     "start_time": "2023-10-26T00:13:21.181692Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "成功获取https://www.hupu.com/home/v1/news?pageNo=1&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=3&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=2&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=4&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=6&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=5&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=7&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=8&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=9&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=10&pageSize=50\n"
     ]
    }
   ],
   "source": [
    "from threading import Thread\n",
    "from fake_useragent import UserAgent\n",
    "import requests\n",
    "from time import sleep\n",
    "from queue import Queue\n",
    "\n",
    "\n",
    "class MyThread(Thread):\n",
    "    def __init__(self):\n",
    "        Thread.__init__(self)\n",
    "\n",
    "    def run(self):\n",
    "        \"\"\"线程执行的函数\"\"\"\n",
    "        \n",
    "        # 如果队列不为空\n",
    "        while not url_queue.empty():\n",
    "            \n",
    "            # 获取 URL地址\n",
    "            url = url_queue.get()\n",
    "            \n",
    "            # 随机生成用户代理\n",
    "            headers = {'User-Agent': UserAgent().chrome}\n",
    "            \n",
    "            # 发送 GET请求\n",
    "            resp = requests.get(url, headers=headers)\n",
    "            \n",
    "            sleep(3)\n",
    "            \n",
    "            if resp.status_code == 200:\n",
    "                print(f'成功获取{url}')\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    \n",
    "    # 创建队列\n",
    "    url_queue = Queue()\n",
    "    \n",
    "    # 为队列添加 URL地址\n",
    "    for i in range(1, 11):\n",
    "        url = f'https://www.hupu.com/home/v1/news?pageNo={i}&pageSize=50'\n",
    "        url_queue.put(url)\n",
    "\n",
    "    # 启动线程\n",
    "    for i in range(3):\n",
    "        t1 = MyThread()\n",
    "        t1.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "1850d462",
   "metadata": {},
   "source": [
    "### 函数包装"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "34ed2d82",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-25T02:01:19.768946Z",
     "start_time": "2023-10-25T02:01:19.765356Z"
    }
   },
   "outputs": [],
   "source": [
    "from threading import Thread\n",
    "\n",
    "def func(args):\n",
    "    pass\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    \n",
    "    # URL队列\n",
    "    info_html = Queue()\n",
    "    \n",
    "    # 创建线程\n",
    "    t1 = Thread(target=func, # 线程要执行的函数\n",
    "                args=(info_html,)) # 传入参数"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "442178d0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T00:10:08.271999Z",
     "start_time": "2023-10-26T00:10:08.259663Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "成功获取https://www.hupu.com/home/v1/news?pageNo=1&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=2&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=3&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=5&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=4&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=6&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=7&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=9&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=8&pageSize=50\n",
      "成功获取https://www.hupu.com/home/v1/news?pageNo=10&pageSize=50\n"
     ]
    }
   ],
   "source": [
    "from threading import Thread\n",
    "from fake_useragent import UserAgent\n",
    "import requests\n",
    "from time import sleep\n",
    "from queue import Queue\n",
    "\n",
    "\n",
    "def spider():\n",
    "\n",
    "    # 如果 URL队列不为空\n",
    "    while not url_queue.empty():\n",
    "\n",
    "        # 获取 URL地址\n",
    "        url = url_queue.get()\n",
    "        \n",
    "        # 随机生成 用户代理\n",
    "        headers = {'User-Agent': UserAgent().chrome}\n",
    "        \n",
    "        # 发送 GET请求\n",
    "        resp = requests.get(url, headers=headers)\n",
    "        \n",
    "        sleep(3)\n",
    "        \n",
    "        # 如果获取成功，则打印\n",
    "        if resp.status_code == 200:\n",
    "            print(f'成功获取{url}')\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    \n",
    "    # 创建队列\n",
    "    url_queue = Queue()\n",
    "    \n",
    "    # 添加 URL地址\n",
    "    for i in range(1, 11):\n",
    "        url = f'https://www.hupu.com/home/v1/news?pageNo={i}&pageSize=50'\n",
    "        url_queue.put(url)\n",
    "\n",
    "    # 启动 3个线程\n",
    "    for i in range(3):\n",
    "        t1 = Thread(target=spider)\n",
    "        t1.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e3e28449",
   "metadata": {},
   "source": [
    "### 线程池"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "d239aafe",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-25T09:35:58.043706Z",
     "start_time": "2023-10-25T09:35:57.016973Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0678314259\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "101112131415171819\n",
      "\n",
      "\n",
      "\n",
      "16\n",
      "\n",
      "\n",
      "\n",
      "\n",
      "\n"
     ]
    }
   ],
   "source": [
    "# 简单往队列中传输线程数\n",
    "import threading\n",
    "import time\n",
    "import queue\n",
    "\n",
    "\n",
    "class Threadingpool():\n",
    "    \"\"\"线程池类\"\"\"\n",
    "    \n",
    "    def __init__(self, max_num=10):\n",
    "        \n",
    "        # 创建一个容量为10 的队列\n",
    "        self.queue = queue.Queue(max_num)\n",
    "        \n",
    "        # 为队列添加 10个线程\n",
    "        for i in range(max_num):\n",
    "            self.queue.put(threading.Thread)\n",
    "\n",
    "    def getthreading(self):\n",
    "        \"\"\"取线程\"\"\"\n",
    "        return self.queue.get()\n",
    "\n",
    "    def addthreading(self):\n",
    "        \"\"\"添加线程\"\"\"\n",
    "        self.queue.put(threading.Thread)\n",
    "\n",
    "\n",
    "def func(p, i):\n",
    "    \"\"\"线程执行的函数：传入线程池、\"\"\"\n",
    "    time.sleep(1)\n",
    "    print(i)\n",
    "    p.addthreading()\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    \n",
    "    # 创建线程池\n",
    "    p = Threadingpool()\n",
    "    \n",
    "    for i in range(20):\n",
    "        # 取线程\n",
    "        thread = p.getthreading()\n",
    "        \n",
    "        # 配置线程\n",
    "        t = thread(target=func, # 要执行的函数\n",
    "                   args=(p, i)) # 要传递的参数\n",
    "        \n",
    "        # 开启线程\n",
    "        t.start()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4d74944d",
   "metadata": {},
   "source": [
    "## 多进程爬虫\n",
    "multiprocessing 是python的多进程管理包，和 threading.Thread 类似\n",
    "\n",
    "### multiprocessing模块\n",
    "\n",
    "在multiprocessing中，通过创建Process对象生成进程，然后调用它的 start() 方法"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "e2299a62",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-25T09:48:59.303621Z",
     "start_time": "2023-10-25T09:48:59.219784Z"
    }
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<string>\", line 1, in <module>\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/spawn.py\", line 116, in spawn_main\n",
      "    exitcode = _main(fd, parent_sentinel)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/spawn.py\", line 126, in _main\n",
      "    self = reduction.pickle.load(from_parent)\n",
      "AttributeError: Can't get attribute 'func1' on <module '__main__' (built-in)>\n"
     ]
    }
   ],
   "source": [
    "\"\"\"jupyter notebook中，下列代码无法被正常执行，请在Pycharm中尝试\"\"\"\n",
    "\n",
    "from multiprocessing import Process\n",
    "\n",
    "def func1(name):\n",
    "    \"\"\"进程执行的函数\"\"\"\n",
    "    print('hello', name)\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    \n",
    "    # 创建进程\n",
    "    p = Process(target=func1,args=('BIT',))\n",
    "    \n",
    "    # 开启进程\n",
    "    p.start()\n",
    "    \n",
    "    # 等待进程执行完毕\n",
    "    p.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "36654bc1",
   "metadata": {},
   "source": [
    "### Manager类\n",
    "\n",
    "在使用并发设计的时候最好 避免共享数据，尤其是在使用多进程的时候。 \n",
    "\n",
    "如果你真有需要 要共享数据，可以使用由 Manager() 返回的manager提供list, dict, Namespace, Lock, RLock, Semaphore, BoundedSemaphore, Condition, Event, Barrier, Queue, Value and Array类型的支持"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "8bc5cbc0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-25T10:05:42.660155Z",
     "start_time": "2023-10-25T10:05:42.496264Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Traceback (most recent call last):\n",
      "  File \"<string>\", line 1, in <module>\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/spawn.py\", line 116, in spawn_main\n",
      "    exitcode = _main(fd, parent_sentinel)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/spawn.py\", line 126, in _main\n",
      "    self = reduction.pickle.load(from_parent)\n",
      "AttributeError: Can't get attribute 'put_num' on <module '__main__' (built-in)>\n",
      "Traceback (most recent call last):\n",
      "  File \"<string>\", line 1, in <module>\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/spawn.py\", line 116, in spawn_main\n",
      "    exitcode = _main(fd, parent_sentinel)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/spawn.py\", line 126, in _main\n",
      "    self = reduction.pickle.load(from_parent)\n",
      "AttributeError: Can't get attribute 'put_num' on <module '__main__' (built-in)>\n",
      "Traceback (most recent call last):\n",
      "  File \"<string>\", line 1, in <module>\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/spawn.py\", line 116, in spawn_main\n",
      "    exitcode = _main(fd, parent_sentinel)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/spawn.py\", line 126, in _main\n",
      "    self = reduction.pickle.load(from_parent)\n",
      "AttributeError: Can't get attribute 'get_num' on <module '__main__' (built-in)>\n"
     ]
    }
   ],
   "source": [
    "\"\"\"jupyter notebook中，下列代码无法被正常执行，请在Pycharm中尝试\"\"\"\n",
    "\n",
    "from multiprocessing import Process, Manager, Lock\n",
    "\n",
    "\n",
    "def put_num(info_queue, l, lo):\n",
    "    \"\"\"添加数字\"\"\"\n",
    "    \n",
    "    # 获得锁\n",
    "    with lo:\n",
    "        \n",
    "        # 向队列添加数字\n",
    "        for n in l:\n",
    "            info_queue.put(n)\n",
    "\n",
    "\n",
    "def get_num(info_queue, lo):\n",
    "    \"\"\"取出数字\"\"\"\n",
    "    \n",
    "    # 获取锁\n",
    "    with lo:\n",
    "        \n",
    "        # 如果队列不为空，则取出数字\n",
    "        while not info_queue.empty():\n",
    "            print(info_queue.get())\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    \n",
    "    # 创建 Manager对象\n",
    "    manager = Manager()\n",
    "    \n",
    "    # 创建 Queue队列\n",
    "    into_html = manager.Queue()\n",
    "    \n",
    "    # 锁对象\n",
    "    lock = Lock()\n",
    "    a = [1, 2, 3, 4, 5]\n",
    "    b = [11, 12, 13, 14, 15]\n",
    "\n",
    "    # 开启第一个进程\n",
    "    p1 = Process(target=put_num, args=(into_html, a, lock))\n",
    "    p1.start()\n",
    "    \n",
    "    # 开启第二个进程\n",
    "    p2 = Process(target=put_num, args=(into_html, b, lock))\n",
    "    p2.start()\n",
    "    \n",
    "    # 开启第三个进程\n",
    "    p3 = Process(target=get_num, args=(into_html, lock))\n",
    "    p3.start()\n",
    "    \n",
    "    # 等待 3个进程完成\n",
    "    p1.join()\n",
    "    p2.join()\n",
    "    p3.join()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "2503fdf6",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"jupyter notebook中，下列代码无法被正常执行，请在Pycharm中尝试\"\"\"\n",
    "from multiprocessing import Process\n",
    "from multiprocessing import Manager\n",
    "import time\n",
    "from fake_useragent import UserAgent\n",
    "import requests\n",
    "from time import sleep\n",
    "\n",
    "\n",
    "def spider(url_queue):\n",
    "    \n",
    "    # 如果 队列不为空\n",
    "    while not url_queue.empty():\n",
    "        try:\n",
    "            # 获取 URL地址\n",
    "            url = url_queue.get(timeout = 1)\n",
    "            \n",
    "            print(f'成功获取{url}')\n",
    "           \n",
    "            sleep(1)\n",
    "           \n",
    "        except Exception as e:\n",
    "            print(e)\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    \n",
    "    # 创建队列\n",
    "    url_queue = Manager().Queue()\n",
    "    \n",
    "    # 添加 URL地址\n",
    "    for i in range(1,11):\n",
    "        url = f'https://www.hupu.com/home/v1/news?pageNo={i}&pageSize=50'\n",
    "        url_queue.put(url)\n",
    "    \n",
    "    # 存储进程信息\n",
    "    all_process = []\n",
    "    \n",
    "    # 开启进程\n",
    "    for i in range(3):\n",
    "        p1 = Process(target=spider,args=(url_queue,))\n",
    "        p1.start()\n",
    "        all_process.append(p1)\n",
    "        \n",
    "    # 等待所有进程结束，主进程再结束\n",
    "    [p.join() for p in all_process]    "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b490eb8d",
   "metadata": {},
   "source": [
    "### 进程池的使用\n",
    "\n",
    "- 进程池内部维护一个进程序列，当使用时，则去进程池中获取一个进程，如果进程池序列中没有可供使用的进程，那么程序就会等待，直到进程池中有可用进程为止。\n",
    "\n",
    "- 进程池中有两个方法：\n",
    "  - apply 同步执行 - 串行\n",
    "  - apply_async 异步执行 - 并行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "d9261365",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-25T10:23:00.150817Z",
     "start_time": "2023-10-25T10:22:47.710399Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Process SpawnPoolWorker-2:\n",
      "Process SpawnPoolWorker-4:\n",
      "Process SpawnPoolWorker-3:\n",
      "Traceback (most recent call last):\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 315, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 108, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/pool.py\", line 114, in worker\n",
      "    task = get()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/queues.py\", line 368, in get\n",
      "    return _ForkingPickler.loads(res)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 315, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 108, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "AttributeError: Can't get attribute 'get_num' on <module '__main__' (built-in)>\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/pool.py\", line 114, in worker\n",
      "    task = get()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/queues.py\", line 368, in get\n",
      "    return _ForkingPickler.loads(res)\n",
      "Traceback (most recent call last):\n",
      "AttributeError: Can't get attribute 'put_num' on <module '__main__' (built-in)>\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 315, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 108, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/pool.py\", line 114, in worker\n",
      "    task = get()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/queues.py\", line 368, in get\n",
      "    return _ForkingPickler.loads(res)\n",
      "AttributeError: Can't get attribute 'put_num' on <module '__main__' (built-in)>\n",
      "Process SpawnPoolWorker-6:\n",
      "Process SpawnPoolWorker-5:\n",
      "Process SpawnPoolWorker-7:\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 315, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 108, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/pool.py\", line 114, in worker\n",
      "    task = get()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/queues.py\", line 365, in get\n",
      "    with self._rlock:\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/synchronize.py\", line 95, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "Traceback (most recent call last):\n",
      "KeyboardInterrupt\n",
      "Traceback (most recent call last):\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 315, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 108, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/pool.py\", line 114, in worker\n",
      "    task = get()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/queues.py\", line 366, in get\n",
      "    res = self._reader.recv_bytes()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/connection.py\", line 221, in recv_bytes\n",
      "    buf = self._recv_bytes(maxlength)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/connection.py\", line 419, in _recv_bytes\n",
      "    buf = self._recv(4)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/connection.py\", line 384, in _recv\n",
      "    chunk = read(handle, remaining)\n",
      "KeyboardInterrupt\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 315, in _bootstrap\n",
      "    self.run()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/process.py\", line 108, in run\n",
      "    self._target(*self._args, **self._kwargs)\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/pool.py\", line 114, in worker\n",
      "    task = get()\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/queues.py\", line 365, in get\n",
      "    with self._rlock:\n",
      "  File \"/Users/macbook/opt/anaconda3/lib/python3.9/multiprocessing/synchronize.py\", line 95, in __enter__\n",
      "    return self._semlock.__enter__()\n",
      "KeyboardInterrupt\n"
     ]
    },
    {
     "ename": "KeyboardInterrupt",
     "evalue": "",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m                         Traceback (most recent call last)",
      "Input \u001b[0;32mIn [2]\u001b[0m, in \u001b[0;36m<cell line: 18>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     34\u001b[0m pool\u001b[38;5;241m.\u001b[39mclose()\n\u001b[1;32m     36\u001b[0m \u001b[38;5;66;03m# 进程池中进程执行完毕后再关闭，如果注释，那么程序直接关闭\u001b[39;00m\n\u001b[0;32m---> 37\u001b[0m \u001b[43mpool\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mjoin\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/opt/anaconda3/lib/python3.9/multiprocessing/pool.py:662\u001b[0m, in \u001b[0;36mPool.join\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    660\u001b[0m \u001b[38;5;28;01melif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_state \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;129;01min\u001b[39;00m (CLOSE, TERMINATE):\n\u001b[1;32m    661\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIn unknown state\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m--> 662\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_worker_handler\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mjoin\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    663\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_task_handler\u001b[38;5;241m.\u001b[39mjoin()\n\u001b[1;32m    664\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_result_handler\u001b[38;5;241m.\u001b[39mjoin()\n",
      "File \u001b[0;32m~/opt/anaconda3/lib/python3.9/threading.py:1053\u001b[0m, in \u001b[0;36mThread.join\u001b[0;34m(self, timeout)\u001b[0m\n\u001b[1;32m   1050\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mcannot join current thread\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m   1052\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m timeout \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[0;32m-> 1053\u001b[0m     \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_wait_for_tstate_lock\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m   1054\u001b[0m \u001b[38;5;28;01melse\u001b[39;00m:\n\u001b[1;32m   1055\u001b[0m     \u001b[38;5;66;03m# the behavior of a negative timeout isn't documented, but\u001b[39;00m\n\u001b[1;32m   1056\u001b[0m     \u001b[38;5;66;03m# historically .join(timeout=x) for x<0 has acted as if timeout=0\u001b[39;00m\n\u001b[1;32m   1057\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_wait_for_tstate_lock(timeout\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mmax\u001b[39m(timeout, \u001b[38;5;241m0\u001b[39m))\n",
      "File \u001b[0;32m~/opt/anaconda3/lib/python3.9/threading.py:1073\u001b[0m, in \u001b[0;36mThread._wait_for_tstate_lock\u001b[0;34m(self, block, timeout)\u001b[0m\n\u001b[1;32m   1070\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m\n\u001b[1;32m   1072\u001b[0m \u001b[38;5;28;01mtry\u001b[39;00m:\n\u001b[0;32m-> 1073\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[43mlock\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43macquire\u001b[49m\u001b[43m(\u001b[49m\u001b[43mblock\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtimeout\u001b[49m\u001b[43m)\u001b[49m:\n\u001b[1;32m   1074\u001b[0m         lock\u001b[38;5;241m.\u001b[39mrelease()\n\u001b[1;32m   1075\u001b[0m         \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_stop()\n",
      "\u001b[0;31mKeyboardInterrupt\u001b[0m: "
     ]
    }
   ],
   "source": [
    "\"\"\"jupyter notebook中，下列代码无法被正常执行，请在Pycharm中尝试\"\"\"\n",
    "\n",
    "from multiprocessing import Pool, Manager\n",
    "\n",
    "\n",
    "def put_num(info_queue, l):\n",
    "    \"\"\"添加数字\"\"\"\n",
    "    for n in l:\n",
    "        info_queue.put(n)\n",
    "\n",
    "\n",
    "def get_num(info_queue):\n",
    "    \"\"\"获取数字\"\"\"\n",
    "    \n",
    "    # 当队列不为空时，获取数字\n",
    "    while not info_queue.empty():\n",
    "        print(info_queue.get())\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "\n",
    "    # 创建 Queue队列\n",
    "    html_queue = Manager().Queue()\n",
    "    a = [11, 12, 13, 14, 15]\n",
    "    b = [1, 2, 3, 4, 5]\n",
    "\n",
    "    # 创建容量为3 的进程池\n",
    "    pool = Pool(3)\n",
    "\n",
    "    # 异步执行\n",
    "    pool.apply_async(func=put_num, args=(html_queue, a))\n",
    "    pool.apply_async(func=put_num, args=(html_queue, b))\n",
    "    pool.apply_async(func=get_num, args=(html_queue,))\n",
    "\n",
    "    # 这里join一定是在close之后，且必须要加join，否则主进程不等待创建的子进程执行完毕\n",
    "    pool.close()\n",
    "\n",
    "    # 进程池中进程执行完毕后再关闭，如果注释，那么程序直接关闭\n",
    "    pool.join()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "5b6faf7c",
   "metadata": {},
   "outputs": [],
   "source": [
    "\"\"\"jupyter notebook中，下列代码无法被正常执行，请在Pycharm中尝试\"\"\"\n",
    "\n",
    "from multiprocessing import Pool, Manager\n",
    "from time import sleep\n",
    "\n",
    "\n",
    "def spider(url_queue):\n",
    "    \n",
    "    # 如果 URL队列不为空\n",
    "    while not url_queue.empty():\n",
    "        try:\n",
    "            \n",
    "            # 获取 URL\n",
    "            url = url_queue.get(timeout=1)\n",
    "            \n",
    "            print(f'成功获取{url}')\n",
    "            \n",
    "            sleep(1)\n",
    "            \n",
    "        except Exception as e:\n",
    "            print(e)\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "\n",
    "    # 创建 队列\n",
    "    url_queue = Manager().Queue()\n",
    "\n",
    "    # 为队列 添加 URL地址\n",
    "    for i in range(1, 11):\n",
    "        url = f'https://www.hupu.com/home/v1/news?pageNo={i}&pageSize=50'\n",
    "        url_queue.put(url)\n",
    "\n",
    "    # 创建 容量为3的进程池\n",
    "    pool = Pool(3)\n",
    "\n",
    "    # 异步执行\n",
    "    pool.apply_async(func=spider, args=(url_queue,))\n",
    "    pool.apply_async(func=spider, args=(url_queue,))\n",
    "    pool.apply_async(func=spider, args=(url_queue,))\n",
    "    \n",
    "    # 关闭进程池\n",
    "    pool.close()\n",
    "    \n",
    "    # 等待子进程结束，主进程才结束\n",
    "    pool.join()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "db345218",
   "metadata": {},
   "source": [
    "## 多协程爬虫"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a82ee42e",
   "metadata": {},
   "source": [
    "### 示例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "69c0c319",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-25T10:22:25.388668Z",
     "start_time": "2023-10-25T10:22:24.951333Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "This event loop is already running",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Input \u001b[0;32mIn [1]\u001b[0m, in \u001b[0;36m<cell line: 24>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     21\u001b[0m loop \u001b[38;5;241m=\u001b[39m asyncio\u001b[38;5;241m.\u001b[39mget_event_loop()\n\u001b[1;32m     23\u001b[0m \u001b[38;5;66;03m# 执行 main函数\u001b[39;00m\n\u001b[0;32m---> 24\u001b[0m \u001b[43mloop\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun_until_complete\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmain\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/opt/anaconda3/lib/python3.9/asyncio/base_events.py:623\u001b[0m, in \u001b[0;36mBaseEventLoop.run_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m    612\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Run until the Future is done.\u001b[39;00m\n\u001b[1;32m    613\u001b[0m \n\u001b[1;32m    614\u001b[0m \u001b[38;5;124;03mIf the argument is a coroutine, it is wrapped in a Task.\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    620\u001b[0m \u001b[38;5;124;03mReturn the Future's result, or raise its exception.\u001b[39;00m\n\u001b[1;32m    621\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    622\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_closed()\n\u001b[0;32m--> 623\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_check_running\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    625\u001b[0m new_task \u001b[38;5;241m=\u001b[39m \u001b[38;5;129;01mnot\u001b[39;00m futures\u001b[38;5;241m.\u001b[39misfuture(future)\n\u001b[1;32m    626\u001b[0m future \u001b[38;5;241m=\u001b[39m tasks\u001b[38;5;241m.\u001b[39mensure_future(future, loop\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m)\n",
      "File \u001b[0;32m~/opt/anaconda3/lib/python3.9/asyncio/base_events.py:583\u001b[0m, in \u001b[0;36mBaseEventLoop._check_running\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    581\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_check_running\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m    582\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mis_running():\n\u001b[0;32m--> 583\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mThis event loop is already running\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m    584\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m events\u001b[38;5;241m.\u001b[39m_get_running_loop() \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    585\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\n\u001b[1;32m    586\u001b[0m             \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mCannot run the event loop while another loop is running\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "\u001b[0;31mRuntimeError\u001b[0m: This event loop is already running"
     ]
    }
   ],
   "source": [
    "import aiohttp\n",
    "import asyncio\n",
    "\n",
    "async def main():\n",
    "    \n",
    "    # 对网址发送 GET请求\n",
    "    async with aiohttp.ClientSession() as session:\n",
    "        async with session.get('http://python.org') as response:\n",
    "            \n",
    "            # 打印 响应码\n",
    "            print(\"Status:\", response.status)\n",
    "            \n",
    "            # 打印 获取的数据类型\n",
    "            print(\"Content-type:\", response.headers['content-type'])\n",
    "\n",
    "            # 获取内容\n",
    "            html = await response.text()\n",
    "            print(\"Body:\", html[:15], \"...\")\n",
    "\n",
    "# 创建 协程\n",
    "loop = asyncio.get_event_loop()\n",
    "\n",
    "# 执行 main函数\n",
    "loop.run_until_complete(main())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "effccca2",
   "metadata": {},
   "source": [
    "### clientSession的使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "affbdf5e",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T00:04:43.295619Z",
     "start_time": "2023-10-26T00:04:42.477263Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"header设置\"\"\"\n",
    "import aiohttp\n",
    "import asyncio\n",
    "\n",
    "# 设置请求头\n",
    "headers = {\"Authorization\": \"Basic bG9naW46cGFzcw==\"}\n",
    "\n",
    "# 发送 GET请求\n",
    "async with aiohttp.ClientSession(headers=headers) as session:\n",
    "    async with session.get(\"http://httpbin.org/headers\") as r:\n",
    "        \n",
    "        # 转成 Json格式\n",
    "        json_body = await r.json()\n",
    "        \n",
    "        # 设置\n",
    "        assert json_body['headers']['Authorization'] == 'Basic bG9naW46cGFzcw=='"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "3589b7dc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T00:04:46.654372Z",
     "start_time": "2023-10-26T00:04:46.069363Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"Cookie的使用\"\"\"\n",
    "\n",
    "url = 'http://httpbin.org/cookies'\n",
    "cookies = {'cookies_are': 'working'}\n",
    "\n",
    "# 设置 cookies，发送 GET请求\n",
    "async with aiohttp.ClientSession(cookies=cookies) as session:\n",
    "    async with session.get(url) as resp:\n",
    "        assert await resp.json() == {\"cookies\": {\"cookies_are\": \"working\"}}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "id": "a8b43aa4",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T00:04:49.444359Z",
     "start_time": "2023-10-26T00:04:48.799486Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "400\n"
     ]
    }
   ],
   "source": [
    "\"\"\"Proxy的使用\"\"\"\n",
    "\n",
    "# 发送 GET请求，设置代理\n",
    "async with aiohttp.ClientSession() as session:\n",
    "    async with session.get(\"http://python.org\", proxy=\"http://proxy.com\") as resp:\n",
    "        \n",
    "        # 获取状态\n",
    "        print(resp.status)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2f39bdee",
   "metadata": {},
   "source": [
    "**Response的使用**\n",
    "\n",
    "- 属性\n",
    "  - status 状态码\n",
    "  - url url地址\n",
    "  - cookies cookie内容\n",
    "  - headers 响应头\n",
    "  \n",
    "- 方法\n",
    "  - read() 读bytes类型\n",
    "  - text() 读文本内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 11,
   "id": "2bcda6dd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T00:28:26.494061Z",
     "start_time": "2023-10-26T00:28:26.434772Z"
    },
    "collapsed": true
   },
   "outputs": [
    {
     "ename": "RuntimeError",
     "evalue": "This event loop is already running",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mRuntimeError\u001b[0m                              Traceback (most recent call last)",
      "Input \u001b[0;32mIn [11]\u001b[0m, in \u001b[0;36m<cell line: 61>\u001b[0;34m()\u001b[0m\n\u001b[1;32m     61\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;18m__name__\u001b[39m \u001b[38;5;241m==\u001b[39m \u001b[38;5;124m'\u001b[39m\u001b[38;5;124m__main__\u001b[39m\u001b[38;5;124m'\u001b[39m:\n\u001b[1;32m     62\u001b[0m     \n\u001b[1;32m     63\u001b[0m     \u001b[38;5;66;03m# 启动协程\u001b[39;00m\n\u001b[1;32m     64\u001b[0m     loop \u001b[38;5;241m=\u001b[39m asyncio\u001b[38;5;241m.\u001b[39mget_event_loop()\n\u001b[0;32m---> 65\u001b[0m     \u001b[43mloop\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mrun_until_complete\u001b[49m\u001b[43m(\u001b[49m\u001b[43mtest_cookie\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/opt/anaconda3/lib/python3.9/asyncio/base_events.py:623\u001b[0m, in \u001b[0;36mBaseEventLoop.run_until_complete\u001b[0;34m(self, future)\u001b[0m\n\u001b[1;32m    612\u001b[0m \u001b[38;5;250m\u001b[39m\u001b[38;5;124;03m\"\"\"Run until the Future is done.\u001b[39;00m\n\u001b[1;32m    613\u001b[0m \n\u001b[1;32m    614\u001b[0m \u001b[38;5;124;03mIf the argument is a coroutine, it is wrapped in a Task.\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    620\u001b[0m \u001b[38;5;124;03mReturn the Future's result, or raise its exception.\u001b[39;00m\n\u001b[1;32m    621\u001b[0m \u001b[38;5;124;03m\"\"\"\u001b[39;00m\n\u001b[1;32m    622\u001b[0m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_check_closed()\n\u001b[0;32m--> 623\u001b[0m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_check_running\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    625\u001b[0m new_task \u001b[38;5;241m=\u001b[39m \u001b[38;5;129;01mnot\u001b[39;00m futures\u001b[38;5;241m.\u001b[39misfuture(future)\n\u001b[1;32m    626\u001b[0m future \u001b[38;5;241m=\u001b[39m tasks\u001b[38;5;241m.\u001b[39mensure_future(future, loop\u001b[38;5;241m=\u001b[39m\u001b[38;5;28mself\u001b[39m)\n",
      "File \u001b[0;32m~/opt/anaconda3/lib/python3.9/asyncio/base_events.py:583\u001b[0m, in \u001b[0;36mBaseEventLoop._check_running\u001b[0;34m(self)\u001b[0m\n\u001b[1;32m    581\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_check_running\u001b[39m(\u001b[38;5;28mself\u001b[39m):\n\u001b[1;32m    582\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mis_running():\n\u001b[0;32m--> 583\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mThis event loop is already running\u001b[39m\u001b[38;5;124m'\u001b[39m)\n\u001b[1;32m    584\u001b[0m     \u001b[38;5;28;01mif\u001b[39;00m events\u001b[38;5;241m.\u001b[39m_get_running_loop() \u001b[38;5;129;01mis\u001b[39;00m \u001b[38;5;129;01mnot\u001b[39;00m \u001b[38;5;28;01mNone\u001b[39;00m:\n\u001b[1;32m    585\u001b[0m         \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mRuntimeError\u001b[39;00m(\n\u001b[1;32m    586\u001b[0m             \u001b[38;5;124m'\u001b[39m\u001b[38;5;124mCannot run the event loop while another loop is running\u001b[39m\u001b[38;5;124m'\u001b[39m)\n",
      "\u001b[0;31mRuntimeError\u001b[0m: This event loop is already running"
     ]
    }
   ],
   "source": [
    "import aiohttp\n",
    "import asyncio\n",
    "\n",
    "\n",
    "async def first():\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    async with aiohttp.ClientSession() as session:  # aiohttp.ClientSession() == import requests 模块\n",
    "        async with session.get('http://httpbin.org/get') as resp:\n",
    "            \n",
    "            # 获取内容\n",
    "            rs = await resp.text()\n",
    "            print(rs[:100])\n",
    "\n",
    "headers = {'User-Agent': 'aaaaaa123'}\n",
    "\n",
    "async def test_header():\n",
    "    \n",
    "    # 设置请求头，发送 GET请求\n",
    "    async with aiohttp.ClientSession(headers=headers) as session:\n",
    "        async with session.get('http://httpbin.org/get') as resp:\n",
    "            \n",
    "            # 获取内容\n",
    "            rs = await resp.text()\n",
    "            print(rs[:100])\n",
    "\n",
    "\n",
    "async def test_params():\n",
    "    \n",
    "    # 设置请求头，发送带参数的 GET请求\n",
    "    async with aiohttp.ClientSession(headers=headers) as session:\n",
    "        async with session.get('http://httpbin.org/get', params={'name': 'bjsxt'}) as resp:\n",
    "            \n",
    "            # 获取内容\n",
    "            rs = await resp.text()\n",
    "            print(rs[:100])\n",
    "\n",
    "\n",
    "async def test_cookie():\n",
    "    \n",
    "    # 设置请求头，添加 cookies，发送带参数的 GET请求\n",
    "    async with aiohttp.ClientSession(headers=headers, cookies={'token': 'sxt123id'}) as session:\n",
    "        async with session.get('http://httpbin.org/get', params={'name': 'bjsxt'}) as resp:\n",
    "            \n",
    "            # 获取内容\n",
    "            rs = await resp.text()\n",
    "            print(rs[:100])\n",
    "\n",
    "\n",
    "async def test_proxy():\n",
    "    \n",
    "    # 设置请求头，添加 cookies与代理，发送带参数的 GET请求\n",
    "    async with aiohttp.ClientSession(headers=headers, cookies={'token': 'sxt123id'}) as session:\n",
    "        async with session.get('http://httpbin.org/get', params={'name': 'bjsxt'}, proxy='http://name:pwd@ip:port') as resp:\n",
    "            \n",
    "            # 获取内容\n",
    "            rs = await resp.text()\n",
    "            print(rs)\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    \n",
    "    # 启动协程\n",
    "    loop = asyncio.get_event_loop()\n",
    "    loop.run_until_complete(test_cookie())"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e0c18202",
   "metadata": {},
   "source": [
    "## Selenium初步使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "id": "4d09b0cc",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T00:53:20.927547Z",
     "start_time": "2023-10-26T00:53:17.865837Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html><head><script type=\"text/javascript\" charset=\"utf-8\" src=\"https://pss.bdstatic.com/static/supe\n",
      "[{'domain': '.baidu.com', 'expiry': 1729817600, 'httpOnly': False, 'name': 'ZFY', 'path': '/', 'sameSite': 'None', 'secure': True, 'value': 'kcGUACtNx2KdcyAwkb5ceh:A3UjfLUmFqpYcyMSLBRvY:C'}, {'domain': '.baidu.com', 'expiry': 1729817600, 'httpOnly': False, 'name': 'BAIDUID_BFESS', 'path': '/', 'sameSite': 'None', 'secure': True, 'value': '47CBFCBBA3EF8A5E6F7B68CA454E422F:FG=1'}, {'domain': '.baidu.com', 'expiry': 1698368000, 'httpOnly': False, 'name': 'BA_HECTOR', 'path': '/', 'sameSite': 'Lax', 'secure': False, 'value': 'ada020a100a5004ha18h20aa1ijje401p'}, {'domain': 'www.baidu.com', 'expiry': 1698281600, 'httpOnly': False, 'name': 'BD_LAST_QID', 'path': '/', 'sameSite': 'Lax', 'secure': False, 'value': '11234071065355764699'}, {'domain': '.baidu.com', 'expiry': 1729817599, 'httpOnly': False, 'name': 'BAIDUID', 'path': '/', 'sameSite': 'Lax', 'secure': False, 'value': '47CBFCBBA3EF8A5E6F7B68CA454E422F:FG=1'}, {'domain': '.baidu.com', 'expiry': 1732841599, 'httpOnly': False, 'name': 'BIDUPSID', 'path': '/', 'sameSite': 'Lax', 'secure': False, 'value': '47CBFCBBA3EF8A5ED89D46B5AFB538E7'}, {'domain': '.baidu.com', 'expiry': 1732841599, 'httpOnly': False, 'name': 'PSTM', 'path': '/', 'sameSite': 'Lax', 'secure': False, 'value': '1698281600'}, {'domain': 'www.baidu.com', 'expiry': 1699145600, 'httpOnly': False, 'name': 'BD_UPN', 'path': '/', 'sameSite': 'Lax', 'secure': False, 'value': '123253'}]\n",
      "https://www.baidu.com/\n"
     ]
    }
   ],
   "source": [
    "# 导入 webdriver\n",
    "from selenium import webdriver\n",
    "\n",
    "# 要想调用键盘按键操作需要引入keys包\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "\n",
    "# 创建 浏览器对象\n",
    "driver = webdriver.Chrome()\n",
    "\n",
    "# 对浏览器发送 GET请求\n",
    "driver.get(\"http://www.baidu.com/\")\n",
    "\n",
    "# 获取网页的源代码\n",
    "print(driver.page_source[:100])\n",
    "\n",
    "# 获取当前页面 Cookie\n",
    "print(driver.get_cookies())\n",
    "\n",
    "# 生成新的页面快照\n",
    "driver.save_screenshot(\"./data/python爬虫.png\")\n",
    "\n",
    "# 获取当前 url\n",
    "print(driver.current_url)\n",
    "\n",
    "# 关闭浏览器\n",
    "driver.quit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 15,
   "id": "e6d96ba8",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T00:57:07.657648Z",
     "start_time": "2023-10-26T00:57:02.625434Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "# 创建 chrome浏览器对象\n",
    "chrome = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "chrome.get('http://www.baidu.com/')\n",
    "\n",
    "# 获取页面源代码\n",
    "html =  chrome.page_source\n",
    "\n",
    "# 获取页面的 URL地址\n",
    "url = chrome.current_url\n",
    "\n",
    "# 保存页面的快照\n",
    "chrome.save_screenshot('./baidu.png')\n",
    "\n",
    "sleep(2)\n",
    "\n",
    "# 退出浏览器\n",
    "chrome.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "b5cad107",
   "metadata": {},
   "source": [
    "## Selenium 控制浏览器"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "387447bb",
   "metadata": {},
   "source": [
    "### 最大化窗口\n",
    "\n",
    "我们知道调用启动的浏览器 不是全屏的，这样不会影响脚本的执行，但是有时候会影响我们“观看”脚本的执行"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 18,
   "id": "44d1b503",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T01:00:10.396756Z",
     "start_time": "2023-10-26T01:00:05.857485Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "# 创建 浏览器对象\n",
    "browser = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "url= 'http://www.baidu.com'\n",
    "browser.get(url)\n",
    "\n",
    "# 最大化窗口\n",
    "browser.maximize_window()\n",
    "\n",
    "sleep(2)\n",
    "\n",
    "# 退出浏览器\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "754c3d1d",
   "metadata": {},
   "source": [
    "### 设置宽与高\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 19,
   "id": "687ed5cb",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T01:01:49.260102Z",
     "start_time": "2023-10-26T01:01:45.136593Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "# 创建 浏览器对象\n",
    "browser = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "url = 'http://www.baidu.com'\n",
    "browser.get(url)\n",
    "\n",
    "# 设置窗口大小\n",
    "browser.set_window_size(500, 600)\n",
    "\n",
    "sleep(2)\n",
    "\n",
    "# 退出浏览器\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "60660ca9",
   "metadata": {},
   "source": [
    "### 浏览器 前进、后退\n",
    "\n",
    "浏览器上有一个后退、前进按钮，对于浏览网页的人是比较方便的；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "815d9cc7",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T01:05:15.167928Z",
     "start_time": "2023-10-26T01:05:04.571286Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "\n",
    "# 创建 浏览器对象\n",
    "browser = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求，访问百度首页\n",
    "first_url = 'http://www.baidu.com'\n",
    "browser.get(first_url)\n",
    "\n",
    "time.sleep(2)\n",
    "\n",
    "# 发送 GET请求，访问新闻页面\n",
    "second_url = 'http://news.baidu.com'\n",
    "browser.get(second_url)\n",
    "\n",
    "time.sleep(2)  \n",
    "\n",
    "# 返回（后退）到 百度首页\n",
    "browser.back()\n",
    "time.sleep(1)  \n",
    " \n",
    "# 前进到 新闻页\n",
    "browser.forward()\n",
    "time.sleep(2)\n",
    "\n",
    "# 退出浏览器\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "70f69a4a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T01:09:57.812640Z",
     "start_time": "2023-10-26T01:09:26.865446Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "\n",
    "\n",
    "def test01():\n",
    "    \n",
    "    # 创建 浏览器对象\n",
    "    chrome = webdriver.Chrome()\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    url = 'http://www.baidu.com'\n",
    "    chrome.get(url)\n",
    "    \n",
    "    # 最大化窗口\n",
    "    chrome.maximize_window()\n",
    "    \n",
    "    sleep(3)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "def test02():\n",
    "    \n",
    "    # 创建 浏览器对象\n",
    "    chrome = webdriver.Chrome()\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    url = 'http://www.baidu.com'\n",
    "    chrome.get(url)\n",
    "    \n",
    "    # 设置浏览器宽与高\n",
    "    chrome.set_window_size(500, 600)  \n",
    "    \n",
    "    sleep(3)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "def test03():\n",
    "    \n",
    "    # 创建 浏览器对象\n",
    "    chrome = webdriver.Chrome()\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    url = 'http://www.baidu.com'\n",
    "    chrome.get(url)\n",
    "    \n",
    "    sleep(2)\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    chrome.get('http://www.bjsxt.com/')\n",
    "    sleep(3)\n",
    "    \n",
    "    # 后退\n",
    "    chrome.back()   \n",
    "    sleep(2)\n",
    "    \n",
    "    # 前进\n",
    "    chrome.forward()    \n",
    "    sleep(3)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    test01()\n",
    "    test02()\n",
    "    test03()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "19a6313a",
   "metadata": {},
   "source": [
    "## Selenium 基本操作"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5a1f1142",
   "metadata": {},
   "source": [
    "对象的定位应该是自动化的核心，要想操作一个对象，首先应该识别这个对象。\n",
    "\n",
    "一个对象就像人一样，他会有各种的特征（属性）\n",
    "\n",
    "### 对象定位\n",
    "\n",
    "webdriver提供了一系列的对象定位方法，常用的有以下几种：\n",
    "\n",
    "\n",
    "利用 By 类来确定哪种选择方式\n",
    "\n",
    "```shell\n",
    "from selenium.webdriver.common.by import By\n",
    "```\n",
    "\n",
    "**By 类的一些属性如下：**\n",
    "\n",
    "- ID = \"id\"\n",
    "- XPATH = \"xpath\"\n",
    "- LINK_TEXT = \"link text\"\n",
    "- PARTIAL_LINK_TEXT = \"partial link text\"\n",
    "- NAME = \"name\"\n",
    "- TAG_NAME = \"tag name\"\n",
    "- CLASS_NAME = \"class name\"\n",
    "- CSS_SELECTOR = \"css selector\"\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "cc04d4b0",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T01:34:43.669953Z",
     "start_time": "2023-10-26T01:34:36.555182Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "from selenium.webdriver.common.by import By\n",
    "\n",
    "# 创建 浏览器\n",
    "browser = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "browser.get(\"http://www.baidu.com\")\n",
    "time.sleep(2)\n",
    "\n",
    "## 百度输入框的定位方式\n",
    "# 通过 id方式定位\n",
    "browser.find_element(by=By.ID, value=\"kw\").send_keys(\"selenium\")\n",
    "\n",
    "\"\"\"\n",
    "# 通过 name方式定位\n",
    "browser.find_element(by=By.NAME, value=\"kw\").send_keys(\"selenium\")\n",
    "\n",
    "# 通过 tag name方式定位\n",
    "browser.find_element(by=By.TAG_NAME, value=\"input\").send_keys(\"selenium\")\n",
    "\n",
    "# 通过 class name 方式定位\n",
    "browser.find_element(by=By.CLASS_NAME, value=\"s_ipt\").send_keys(\"selenium\")\n",
    "\n",
    "# 通过 CSS方式定位\n",
    "browser.find_element(by=By.CSS_SELECTOR, value=\"#kw\").send_keys(\"selenium\")\n",
    "\n",
    "# 通过 xphan方式定位\n",
    "browser.find_element(by=By.XPATH, value=\"//input[@id='kw']\").send_keys(\"selenium\")\n",
    "\"\"\"\n",
    "\n",
    "# 定位搜索按钮，并进行点击\n",
    "browser.find_element(by=By.ID, value=\"su\").click()\n",
    "\n",
    "\n",
    "time.sleep(3)\n",
    "\n",
    "# 退出浏览器\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2dd808c8",
   "metadata": {},
   "source": [
    "### 层级定位\n",
    "\n",
    "在实际的开发中也经常会遇到这种问题：页面上有很多个属性基本相同的元素，现在需要具体定位到其中的一个。\n",
    "\n",
    "由于属性基本相当，所以在定位的时候会有些麻烦，这时候就需要用到层级定位。先定位父元素，然后再通过父元素定位子孙元素"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f0f14501",
   "metadata": {},
   "source": [
    "**具体思路是：** 先点击显示出 1个下拉菜单，然后再定位到该下拉菜单所在的ul，再定位这个ul下的某个具体的link。在这里，我们定位第1个下拉菜单中的Action这个选项。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "dc2e472f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T01:50:20.104584Z",
     "start_time": "2023-10-26T01:50:15.392900Z"
    },
    "scrolled": true
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "file:////Users/macbook/Documents/Python全栈/06_Python爬虫/html/level_locate.html\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.support.ui import WebDriverWait\n",
    "from selenium.webdriver.common.by import By\n",
    "import time\n",
    "import os\n",
    "\n",
    "# 创建 浏览器对象\n",
    "chrome = webdriver.Chrome()\n",
    "\n",
    "# 获取 HTML文件的绝对路径\n",
    "file_path = 'file:///' + os.path.abspath('./html/level_locate.html')\n",
    "print(file_path)\n",
    "\n",
    "# 发送 GET请求\n",
    "chrome.get(file_path)\n",
    "\n",
    "# 点击 Link1链接（弹出下拉列表)\n",
    "chrome.find_element(by=By.LINK_TEXT, value='Link1').click()\n",
    "\n",
    "# 找到 id 为 dropdown1的父元素\n",
    "# WebDriverWait(chrome, 10)：10秒内每隔 500毫秒扫描 1次页面变化，当出现指定的元素后结束\n",
    "WebDriverWait(chrome, 10).until(\n",
    "    lambda the_driver: the_driver.find_element(by=By.ID, value='dropdown1').is_displayed())\n",
    "\n",
    "# 在父亲元件下找到link为Action的子元素\n",
    "menu = chrome.find_element(by=By.ID, value='dropdown1').find_element(by=By.LINK_TEXT,value='Action')\n",
    "\n",
    "# 鼠标定位到子元素上\n",
    "webdriver.ActionChains(chrome).move_to_element(menu).perform()\n",
    "\n",
    "time.sleep(2)\n",
    "chrome.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 45,
   "id": "90d13530",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T02:04:31.640554Z",
     "start_time": "2023-10-26T02:04:10.201492Z"
    }
   },
   "outputs": [],
   "source": [
    "\"\"\"定位元素\"\"\"\n",
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "import os\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "\n",
    "\n",
    "def test01():\n",
    "\n",
    "    # 创建 浏览器\n",
    "    chrome = webdriver.Chrome()\n",
    "\n",
    "    # 发送 GET请求\n",
    "    chrome.get('http://www.baidu.com')\n",
    "    sleep(2)\n",
    "\n",
    "    # 查找输入框元素，输入 python\n",
    "    chrome.find_element(by=By.ID, value='kw').send_keys('python')\n",
    "    sleep(2)\n",
    "\n",
    "    # 找到 搜索键，并点击\n",
    "    chrome.find_element(by=By.ID, value='su').click()\n",
    "    sleep(2)\n",
    "\n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "def test02():\n",
    "    \n",
    "    # 创建 浏览器\n",
    "    chrome = webdriver.Chrome()\n",
    "\n",
    "    # 查找 文件绝对路径\n",
    "    file_path = 'file:///'+os.path.abspath('./html/test01.html')\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    chrome.get(file_path)\n",
    "    \n",
    "    # 定位元素，并点击\n",
    "    chrome.find_element(by=By.LINK_TEXT, value='Link1').click()\n",
    "    \n",
    "    # 定位元素\n",
    "    menu = chrome.find_element(by=By.ID, value='dropdown1').find_element(by=By.LINK_TEXT, value='Action')\n",
    "\n",
    "    # webdriver.ActionChains(chrome).move_to_element(menu).perform()    # 移动焦点到某个元素\n",
    "    \n",
    "    # 点击\n",
    "    menu.click()\n",
    "\n",
    "    sleep(2)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    test01()\n",
    "    test02()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "722d912b",
   "metadata": {},
   "source": [
    "### 操作元素\n",
    "\n",
    "定位元素 只是第一步，定位之后需要对这个 元素 进行操作。\n",
    "\n",
    "鼠标点击 还是 键盘输入，这要取决于我们定位的是按钮还输入框。\n",
    "\n",
    "一般来说，webdriver 中比较常用的操作对象的方法有下面几个\n",
    "\n",
    "- click 点击对象\n",
    "- send_keys 在对象上模拟按键输入\n",
    "- clear 清除对象的内容"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "f63b9f19",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T02:11:25.817701Z",
     "start_time": "2023-10-26T02:11:21.698055Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "import os\n",
    "from selenium.webdriver.common.by import By\n",
    "from selenium.webdriver.common.keys import Keys\n",
    "\n",
    "# 创建 浏览器\n",
    "browser = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "browser.get(\"http://www.baidu.com\")\n",
    "\n",
    "# 清空内容\n",
    "browser.find_element(by=By.ID, value=\"kw\").clear()\n",
    "\n",
    "# 键盘输入\n",
    "browser.find_element(by=By.ID, value=\"kw\").send_keys(\"BIT\")\n",
    "\n",
    "# 点击\n",
    "browser.find_element(by=By.ID, value=\"su\").click()\n",
    "\n",
    "sleep(2)\n",
    "\n",
    "# 退出浏览器\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "063e706d",
   "metadata": {},
   "source": [
    "### 多层窗口的定位\n",
    "\n",
    "对于一个现代的web应用，经常会出现框架（frame） 或 窗口（window）的应用，这也就给我们的定位带来了一个难题。\n",
    "\n",
    "有时候我们定位一个元素，定位器没有问题，但一直定位不了，这时候就要检查这个元素是否在一个frame中，seelnium webdriver 提供了一个switch_to_frame方法，可以很轻松的来解决这个问题\n",
    "\n",
    "多层框架或窗口的定位：\n",
    "\n",
    "- switch_to_frame()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 60,
   "id": "8ea0892f",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T02:40:45.054799Z",
     "start_time": "2023-10-26T02:40:39.596725Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "import time \n",
    "import os\n",
    "\n",
    "# 创建 浏览器\n",
    "browser = webdriver.Chrome()\n",
    "\n",
    "# 获取 文件的绝对路径\n",
    "file_path =  'file:///' + os.path.abspath('./html/frame.html')\n",
    "\n",
    "# 发送 GET请求\n",
    "browser.get(file_path)\n",
    "\n",
    "browser.implicitly_wait(30)\n",
    "\n",
    "# 先找到框架（id = f1）\n",
    "browser.switch_to.frame(\"f1\")\n",
    "browser.switch_to.frame(\"f2\")\n",
    "\n",
    "# 下面开始正常地 查找元素\n",
    "browser.find_element(by=By.ID, value=\"kw\").send_keys(\"selenium\")\n",
    "browser.find_element(by=By.ID, value=\"su\").click()\n",
    "time.sleep(3)\n",
    "\n",
    "# 退出浏览器\n",
    "browser.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "76e59891",
   "metadata": {},
   "source": [
    "### selenium 处理下拉框\n",
    "\n",
    "下拉框是我们最常见的一种页面元素，对于一般的元素，我们只需要一次就定位，\n",
    "\n",
    "但下拉框里的内容需要进行两次定位，先定位到下拉框，再定位到下拉框内里的选项。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 62,
   "id": "69ef3d26",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T02:46:38.389981Z",
     "start_time": "2023-10-26T02:46:31.821043Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "from selenium.webdriver.common.by import By\n",
    "import time\n",
    "import os\n",
    "\n",
    "# 创建 浏览器\n",
    "driver = webdriver.Chrome()\n",
    "\n",
    "# 获取文件路径\n",
    "file_path = 'file:///' + os.path.abspath('./html/drop_down.html')\n",
    "\n",
    "# 发送 GET请求\n",
    "driver.get(file_path)\n",
    "\n",
    "time.sleep(2)\n",
    "\n",
    "# 定位下拉框元素\n",
    "m = driver.find_element(by=By.ID, value=\"ShippingMethod\")\n",
    "\n",
    "# 定位选项\n",
    "m.find_element(by=By.XPATH, value=\"//option[@value='10.69']\").click()\n",
    "time.sleep(3)\n",
    "\n",
    "# 退出浏览器\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "93954afe",
   "metadata": {},
   "source": [
    "### selenium 处理弹窗"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 63,
   "id": "79eabc1d",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T02:50:24.766269Z",
     "start_time": "2023-10-26T02:50:18.607037Z"
    }
   },
   "outputs": [],
   "source": [
    "from selenium import webdriver\n",
    "import os\n",
    "from time import sleep\n",
    "\n",
    "# 创建 浏览器\n",
    "chrome = webdriver.Chrome()\n",
    "\n",
    "# 获取 文件\n",
    "file = 'file:///' + os.path.abspath('./html/alter.html')\n",
    "\n",
    "# 发送 GET请求\n",
    "chrome.get(file)\n",
    "sleep(2)\n",
    "\n",
    "# 切换至弹窗，并同意\n",
    "chrome.switch_to.alert.accept() \n",
    "sleep(2)\n",
    "\n",
    "# 退出浏览器\n",
    "chrome.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "a2de0d61",
   "metadata": {},
   "source": [
    "### Selenium 调用js方法\n",
    "\n",
    "```python\n",
    "execute_script(script, *args)\n",
    "```\n",
    "\n",
    "有时候我们需要控制页面滚动条上的滚动条，但滚动条 并非页面上的元素，这个时候就需要借助js是来进行操作。\n",
    "\n",
    "一般用到操作滚动条的会两个场景：\n",
    "\n",
    "1. 要操作的页面元素不在当前页面范围，无法进行操作，需要拖动滚动条\n",
    "2. 注册时的法律条文 需要阅读，判断用户是否阅读的标准是：滚动条是否拉到最下方。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "98b8df82",
   "metadata": {},
   "source": [
    "**滚动条回到顶部：**\n",
    "\n",
    "```python\n",
    "js=\"var q=document.getElementById('id').scrollTop=0\"\n",
    "driver.execute_script(js)\n",
    "```\n",
    "\n",
    "**滚动条拉到底部：**\n",
    "\n",
    "```python\n",
    "js=\"var q=document.documentElement.scrollTop=10000\"\n",
    "driver.execute_script(js)\n",
    "```\n",
    "\n",
    "可以修改scrollTop 的值，来定位右侧滚动条的位置，0是最上面，10000是最底部\n",
    "\n",
    "以上方法在Firefox和IE浏览器上是可以的，但是用Chrome浏览器，发现不管用。Chrome浏览器解决办法：\n",
    "\n",
    "```python\n",
    "js = \"var q=document.body.scrollTop=0\"\n",
    "driver.execute_script(js)\n",
    "```\n",
    "\n",
    "**横向滚动条：**\n",
    "\n",
    "```python\n",
    "js = \"window.scrollTo(100,400)\"\n",
    "driver.execute_script(js)\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 67,
   "id": "dcd5ffd6",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T03:00:35.385171Z",
     "start_time": "2023-10-26T03:00:23.251469Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "30\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "from lxml import etree\n",
    "import time\n",
    "\n",
    "# 加载浏览器\n",
    "wd = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "url = \"https://search.jd.com/Search?keyword=%E7%AC%94%E8%AE%B0%E6%9C%AC&enc=utf-8&wq=%E7%AC%94%E8%AE%B0%E6%9C%AC&pvid=845d019c94f6476ca5c4ffc24df6865a\"\n",
    "wd.get(url)\n",
    "time.sleep(3)\n",
    "\n",
    "# 要执行的 js\n",
    "js = \"var q = document.documentElement.scrollTop=10000\"\n",
    "# 执行js\n",
    "wd.execute_script(js)\n",
    "\n",
    "time.sleep(3)\n",
    "\n",
    "# 解析 HTML数据\n",
    "e = etree.HTML(wd.page_source)\n",
    "\n",
    "# 提取数据的 xpath\n",
    "price_xpath = '//ul[@class=\"gl-warp clearfix\"]//div[@class=\"p-price\"]/strong/i/text()'\n",
    "\n",
    "# 提取数据\n",
    "infos = e.xpath(price_xpath)\n",
    "\n",
    "print(len(infos))\n",
    "time.sleep(3)\n",
    "\n",
    "# 关闭浏览器\n",
    "wd.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2651775b",
   "metadata": {},
   "source": [
    "### selenium 等待处理\n",
    "\n",
    "**强制等待：**\n",
    "\n",
    "> 使用 time.sleep\n",
    "\n",
    "作用：\n",
    "\n",
    "当代码运行到强制等待这一行的时候，无论出于什么原因，都强制等待指定的时间，需要通过time模块实现\n",
    "\n",
    "优点：简单\n",
    "\n",
    "缺点：无法做有效的判断，会浪费时间\n",
    "\n",
    "**隐式等待：**\n",
    "\n",
    "> 到了一定的时间发现元素还没有加载，则继续等待我们指定的时间，如果超过了我们指定的时间还没有加载就会抛出异常，如果没有需要等待的时候就已经加载完毕，就会立即执行\n",
    "\n",
    "优点： 设置一次即可\n",
    "\n",
    "缺点：必须等待加载完成才能到后续的操作，或者等待超时才能进入后续的操作\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 70,
   "id": "010d078c",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T03:09:22.651200Z",
     "start_time": "2023-10-26T03:09:17.678442Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html style=\"--bubble-width: 108px; --bubble-padding-left: 132px;\"><head><meta http-equiv=\"Content-T\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "\n",
    "\n",
    "# 创建 浏览器\n",
    "driver = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "url = 'https://www.baidu.com/'\n",
    "driver.get(url)\n",
    "\n",
    "# 隐式等待：如果查找到我们要的数据，则不再等待\n",
    "driver.implicitly_wait(3)\n",
    "\n",
    "try:\n",
    "    # 查找元素\n",
    "    print(driver.find_element(by=By.CLASS_NAME, value='next'))\n",
    "except:\n",
    "    pass\n",
    "\n",
    "# 打印 源代码\n",
    "print(driver.page_source[:100])\n",
    "\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e2eed012",
   "metadata": {},
   "source": [
    "\n",
    "**显示等待：**\n",
    "\n",
    "> 指定一个等待条件，并且指定一个最长等待时间，会在这个时间内进行判断是否满足等待条件，如果成立就会立即返回，如果不成立，就会一直等待，直到等待你指定的最长等待时间，如果还是不满足，就会抛出异常，如果满足了就会正常返回\n",
    "\n",
    "优点：专门用于对指定一个元素等待，加载完即可运行后续代码\n",
    "\n",
    "缺点：多个元素都需要要单独设置等待\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 72,
   "id": "7e843c34",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T03:12:26.751618Z",
     "start_time": "2023-10-26T03:12:21.792997Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "元素获取失败\n",
      "<html lang=\"en\" data-n-head=\"%7B%22lang%22:%7B%22ssr%22:%22en%22%7D%7D\" class=\"webp\"><head>\n",
      "    <tit\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "\n",
    "# 创建 浏览器\n",
    "driver = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "url = 'https://www.guazi.com/nj/buy/'\n",
    "driver.get(url)\n",
    "\n",
    "# 显示等待\n",
    "wait = WebDriverWait(driver, 10, 0.5)\n",
    "\n",
    "try:\n",
    "    wait.until(EC.presence_of_element_located((By.CLASS_NAME, 'next')))\n",
    "    \n",
    "except:\n",
    "    print('元素获取失败')\n",
    "    \n",
    "print(driver.page_source[:100])\n",
    "\n",
    "driver.quit()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 78,
   "id": "80b1b81a",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T03:21:17.778201Z",
     "start_time": "2023-10-26T03:21:05.089952Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "test01 查找指定元素失败！\n",
      "test02 查找指定元素失败！\n",
      "test03 查找指定元素失败！\n"
     ]
    }
   ],
   "source": [
    "\"\"\"处理等待的代码\"\"\"\n",
    "from selenium import webdriver\n",
    "from time import sleep\n",
    "from selenium.webdriver.support.wait import WebDriverWait\n",
    "\n",
    "\n",
    "def test01():\n",
    "    \n",
    "    # 创建 浏览器\n",
    "    chrome = webdriver.Chrome()\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    chrome.get('https://search.jd.com/Search?keyword=imac&enc=utf-8&wq=imac')\n",
    "\n",
    "    # 执行JS代码 -- 拉动下拉框\n",
    "    js = 'document.documentElement.scrollTop=10000'\n",
    "    chrome.execute_script(js)\n",
    "    \n",
    "    # 隐式等待\n",
    "    chrome.implicitly_wait(2)\n",
    "    \n",
    "    try:\n",
    "        # 查找元素\n",
    "        info = chrome.find_elements_by_xpath('//div[@class=\"p-name p-name-type-2\"]/a/em')\n",
    "        print('test01 查找指定元素成功！')\n",
    "    except:\n",
    "        print('test01 查找指定元素失败！')\n",
    "        \n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "def test02():\n",
    "    \n",
    "    # 创建浏览器对象\n",
    "    chrome = webdriver.Chrome()\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    chrome.get('http://www.baidu.com/')\n",
    "    \n",
    "    # 找到搜索栏，键盘录入\n",
    "    chrome.find_element(by=By.ID, value='kw').send_keys('离岸金融')\n",
    "    \n",
    "    # 点击查询\n",
    "    chrome.find_element(by=By.ID, value='su').click()\n",
    "    \n",
    "    # 隐式等待\n",
    "    chrome.implicitly_wait(2)\n",
    "    \n",
    "    try:\n",
    "        # 查找元素\n",
    "        chrome.find_element_by_xpath('//div[@id=\"1\"]/h3/a').click()\n",
    "        print('test02 查找指定元素成功！')\n",
    "    except:\n",
    "        print('test02 查找指定元素失败！')\n",
    "    \n",
    "    # 强制等待\n",
    "    sleep(3)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "def test03():\n",
    "    \n",
    "    # 创建 浏览器对象\n",
    "    chrome = webdriver.Chrome()\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    chrome.get('http://www.baidu.com/')\n",
    "    \n",
    "    # 找到搜索栏，键盘录入\n",
    "    chrome.find_element(by=By.ID, value='kw').send_keys('离岸金融')\n",
    "    \n",
    "    # 点击查询\n",
    "    chrome.find_element(by=By.ID, value='su').click()\n",
    "    \n",
    "    try:\n",
    "        # 查找元素\n",
    "        chrome.find_element_by_xpath('//div[@id=\"1\"]/h3/a').click()\n",
    "        print('test03 查找指定元素成功！')\n",
    "    except:\n",
    "        print('test03 查找指定元素失败！')\n",
    "\n",
    "    # 强制等待\n",
    "    sleep(3)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    test01()\n",
    "    test02()\n",
    "    test03()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cf766ff2",
   "metadata": {},
   "source": [
    "### 元素拖拽\n",
    "\n",
    "> 要完成元素的拖拽，首先你需要指定被拖动的元素和拖动目标元素，然后利用 ActionChains 类来实现"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 80,
   "id": "3d6128cd",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T03:29:33.920415Z",
     "start_time": "2023-10-26T03:29:09.189530Z"
    }
   },
   "outputs": [],
   "source": [
    "from time import sleep\n",
    "from selenium.webdriver import ActionChains\n",
    "import os\n",
    "\n",
    "# 创建 浏览器\n",
    "chrome = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "chrome.get(f'file:///{os.path.abspath(\"./html/scroll.html\")}')\n",
    "\n",
    "# 查找元素\n",
    "div1 = chrome.find_element(by=By.ID, value='draggable')\n",
    "div2 = chrome.find_element(by=By.ID, value='draggable2')\n",
    "div3 = chrome.find_element(by=By.ID, value='draggable3')\n",
    "\n",
    "# 强制暂停\n",
    "sleep(2)\n",
    "\n",
    "# 创建 ActionChains\n",
    "action_chains = ActionChains(chrome)\n",
    "\n",
    "# 将 div1元素 拖拽至 div2\n",
    "action_chains.drag_and_drop(div1, div2).perform()\n",
    "\n",
    "# 将 div3，向右下拖动10个像素，共拖动 5次\n",
    "a = action_chains.drag_and_drop_by_offset(div3, 10, 10) # 定义动作\n",
    "\n",
    "# 循环执行动作\n",
    "for i in range(5):\n",
    "    a.perform() \n",
    "    sleep(2)\n",
    "    \n",
    "chrome.quit()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eaa2a9df",
   "metadata": {},
   "source": [
    "### 参数的使用"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 84,
   "id": "a68bd004",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T03:44:15.450356Z",
     "start_time": "2023-10-26T03:43:56.750420Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<html><head><script type=\"text/javascript\" charset=\"utf-8\" src=\"https://pss.bdstatic.com/static/supe\n",
      "<html dir=\"ltr\" lang=\"zh\"><head>\n",
      "  <meta charset=\"utf-8\">\n",
      "  <meta name=\"color-scheme\" content=\"light\n",
      "<html><head><script type=\"text/javascript\" charset=\"utf-8\" src=\"https://pss.bdstatic.com/static/supe\n"
     ]
    }
   ],
   "source": [
    "from selenium import webdriver\n",
    "import time\n",
    "\n",
    "def headless():\n",
    "    \"\"\"无头模式\"\"\"\n",
    "    \n",
    "    # 设置成无头浏览器\n",
    "    options = webdriver.ChromeOptions()\n",
    "    options.add_argument('--headless')\n",
    "\n",
    "    # 创建 浏览器\n",
    "    chrome = webdriver.Chrome(options=options)\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    chrome.get('http://www.baidu.com/')\n",
    "    \n",
    "    # 打印 源码\n",
    "    print(chrome.page_source[:100])\n",
    "    \n",
    "    time.sleep(2)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "def proxy():\n",
    "    \"\"\"代理模式\"\"\"\n",
    "    \n",
    "    # 设置代理\n",
    "    options = webdriver.ChromeOptions()\n",
    "    \n",
    "    # options.add_argument('--proxy-server=type://ip:port')\n",
    "    options.add_argument('--proxy-server=http://113.238.142.208:3128')\n",
    "\n",
    "    # 创建 浏览器\n",
    "    chrome = webdriver.Chrome(options=options)\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    chrome.get('http://httpbin.org/get')\n",
    "    \n",
    "    # 打印 源代码\n",
    "    print(chrome.page_source[:100])\n",
    "    time.sleep(2)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "\n",
    "def jiance():\n",
    "    \"\"\"防检测设置\"\"\"\n",
    "    \n",
    "    # 设置防检测\n",
    "    options = webdriver.ChromeOptions()\n",
    "    options.add_experimental_option('excludeSwitches', ['enable-automation'])\n",
    "    options.add_experimental_option('useAutomationExtension', False)\n",
    "\n",
    "    # 创建 浏览器\n",
    "    chrome = webdriver.Chrome()\n",
    "\n",
    "    chrome.execute_cdp_cmd(\"Page.addScriptToEvaluateOnNewDocument\", {\n",
    "        \"source\": \"\"\"\n",
    "        Object.defineProperty(navigator, 'webdriver', {\n",
    "          get: () => false\n",
    "        })\n",
    "      \"\"\"\n",
    "    })\n",
    "\n",
    "    # 发送 GET请求\n",
    "    chrome.get('http://www.baidu.com/')\n",
    "    \n",
    "    # 打印 源代码\n",
    "    print(chrome.page_source[:100])\n",
    "    time.sleep(2)\n",
    "    \n",
    "    # 退出浏览器\n",
    "    chrome.quit()\n",
    "\n",
    "if __name__ == '__main__':\n",
    "    headless()\n",
    "    proxy()\n",
    "    jiance()"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d1f442c4",
   "metadata": {},
   "source": [
    "## Selenium案例"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 88,
   "id": "bcfe0f40",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T03:50:53.590462Z",
     "start_time": "2023-10-26T03:50:48.959878Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "=========第1页数据==========\n",
      "120\n",
      "120\n",
      "120\n"
     ]
    }
   ],
   "source": [
    "\"\"\"遍历虎牙直播的各个页面\"\"\"\n",
    "from selenium import webdriver\n",
    "from lxml import etree\n",
    "\n",
    "# 创建浏览器\n",
    "chrome = webdriver.Chrome()\n",
    "\n",
    "# 发送 GET请求\n",
    "chrome.get('https://www.huya.com/g/lol')\n",
    "\n",
    "# 隐式等待\n",
    "chrome.implicitly_wait(10)\n",
    "\n",
    "i = 0\n",
    "while True:\n",
    "    \n",
    "    # 解析 HTML\n",
    "    e = etree.HTML(chrome.page_source)\n",
    "    \n",
    "    # 1. 使用浏览器的查找元素获取 标签，再获取里面内容\n",
    "    rooms = chrome.find_elements(by=By.CLASS_NAME, value='title') \n",
    "    names = chrome.find_elements(by=By.CLASS_NAME, value='nick')\n",
    "    counts = chrome.find_elements(by=By.CLASS_NAME, value='js-num')\n",
    "\n",
    "    # 2. 直接使用源码，通过源码再用xpath提取 \n",
    "    rooms = e.xpath('//a[@class=\"title\"]/text()')\n",
    "    names = e.xpath('//i[@class=\"nick\"]/text()')\n",
    "    counts = e.xpath('//i[@class=\"js-num\"]/text()')\n",
    "    \n",
    "    print(f'=========第{i+1}页数据==========')\n",
    "    print(len(rooms))\n",
    "    print(len(names))\n",
    "    print(len(counts))\n",
    "    \n",
    "    # 查找元素\n",
    "    has_next = chrome.page_source.find('laypage_next')\n",
    "    \n",
    "    # 如果找到\n",
    "    if has_next != -1:\n",
    "        try:\n",
    "            chrome.find_element(by=By.CLASS_NAME, value='laypage_next').click()\n",
    "        except:\n",
    "            break\n",
    "    else:\n",
    "        break\n",
    "        \n",
    "    i += 1\n",
    "    \n",
    "# 退出浏览器\n",
    "chrome.quit()\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 95,
   "id": "f9dee889",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T04:06:56.864130Z",
     "start_time": "2023-10-26T04:06:50.733021Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "<!DOCTYPE html PUBLIC \"-//W3C//DTD XHTML 1.0 Transitional//EN\" \"http://www.w3.org/TR/xhtml1/DTD/xhtm\n"
     ]
    }
   ],
   "source": [
    "\"\"\"验证码登录1\"\"\"\n",
    "import requests\n",
    "import time\n",
    "\n",
    "\n",
    "def login_input():\n",
    "    \"\"\"输入验证码进行登录\"\"\"\n",
    "    \n",
    "    # 设置请求头\n",
    "    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}\n",
    "\n",
    "    login_url = 'http://www.chaojiying.com/user/login/'\n",
    "    \n",
    "    # 生成验证码的 URL\n",
    "    img_url = 'http://www.chaojiying.com/include/code/code.php?u=1'\n",
    "    \n",
    "    # 发送 GET请求\n",
    "    session = requests.Session()\n",
    "    img_resp = session.get(img_url, headers=headers)\n",
    "    \n",
    "    \n",
    "    # 读取网址中的 验证码图片，并保存在本地\n",
    "    with open('./data/tmp.png', 'wb') as f:\n",
    "        f.write(img_resp.content)\n",
    "\n",
    "    code = input('请输入验证码：')\n",
    "    \n",
    "    # 设置参数\n",
    "    params = {\n",
    "        'user': 'feelingslw',\n",
    "        'pass': '123456',\n",
    "        'imgtxt': code,\n",
    "        'act': '1'\n",
    "    }\n",
    "\n",
    "    # 发送 POST请求\n",
    "    resp = session.post(login_url, data=params, headers=headers)\n",
    "    \n",
    "    # 获取网页内容\n",
    "    print(resp.text[:100])\n",
    "\n",
    "\n",
    "def login_auto():\n",
    "    \"\"\"自动化登录\"\"\"\n",
    "    from data.chaojiying_Python.chaojiying import get_code\n",
    "    \n",
    "    # 设置请求头\n",
    "    headers = {'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}\n",
    "    \n",
    "    # 创建 session对象\n",
    "    session = requests.Session()\n",
    "\n",
    "    login_url = 'http://www.chaojiying.com/user/login/'\n",
    "    img_url = 'http://www.chaojiying.com/include/code/code.php?u=1'\n",
    "\n",
    "    # 发送 GET请求，获取验证码图片\n",
    "    img_resp = session.get(img_url, headers=headers)\n",
    "    \n",
    "    # 读取网址中的 验证码图片，并保存在本地\n",
    "    with open('./data/tmp.png', 'wb') as f:\n",
    "        f.write(img_resp.content)\n",
    "\n",
    "    # 提取验证码\n",
    "    code = get_code('./data/tmp.png')\n",
    "    \n",
    "    # 设置参数\n",
    "    params = {\n",
    "        'user': 'feelingslw',\n",
    "        'pass': '123456',\n",
    "        'imgtxt': code,\n",
    "        'act': '1'\n",
    "    }\n",
    "\n",
    "    # 发送 POST请求\n",
    "    resp = session.post(login_url, data=params, headers=headers)\n",
    "    print(resp.text[:100])\n",
    "\n",
    "\n",
    "if __name__ == \"__main__\":\n",
    "    login_auto()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 98,
   "id": "46707776",
   "metadata": {
    "ExecuteTime": {
     "end_time": "2023-10-26T04:10:46.342475Z",
     "start_time": "2023-10-26T04:10:46.132352Z"
    }
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[]\n",
      "<!DOCTYPE html>\n",
      "<html lang=\"en\">\n",
      "  <head>\n",
      "    <meta charset=\"UTF-8\" />\n",
      "    <meta http-equiv=\"X-UA-Co\n"
     ]
    }
   ],
   "source": [
    "\"\"\"验证码登录2\"\"\"\n",
    "import requests\n",
    "from data.chaojiying_Python.chaojiying import get_code\n",
    "from re import findall\n",
    "\n",
    "login_url = 'https://passport.ganji.com/login.php'\n",
    "img_url = 'https://passport.ganji.com/ajax.php?dir=captcha&module=login_captcha'\n",
    "\n",
    "# 设置请求头\n",
    "headers = {\n",
    "    'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/91.0.4472.124 Safari/537.36'}\n",
    "\n",
    "\n",
    "# 发送 GET请求，打开登录页面\n",
    "session = requests.Session()\n",
    "login_html = session.get(login_url, headers=headers)\n",
    "\n",
    "# 提取 hash\n",
    "hash = findall('\"__hash__\":\"(.+?)\"', login_html.text)\n",
    "\n",
    "# 发送 GET请求，获取验证码\n",
    "img_resp = session.get(img_url, headers=headers)\n",
    "\n",
    "# 读取验证码图片\n",
    "with open('./data/tmp.png', 'wb') as f:\n",
    "    f.write(img_resp.content)\n",
    "\n",
    "# 识别验证码\n",
    "code = get_code('./data/tmp.png')\n",
    "\n",
    "# 构造登录参数\n",
    "data = {\n",
    "    \"username\": \"17703181473\",\n",
    "    \"password\": \"123456qwe\",\n",
    "    \"setcookie\": \"0\",\n",
    "    \"checkCode\": code,\n",
    "    \"next\": \"/\",\n",
    "    \"source\": \"passport\",\n",
    "    \"__hash__\": hash,\n",
    "}\n",
    "\n",
    "# 发送 POST请求，进行登录\n",
    "login_resp = session.post(login_url, headers=headers, data=data)\n",
    "\n",
    "# 打印结果\n",
    "print(login_resp.text[:100])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "c8d892c2",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "54e7051b",
   "metadata": {},
   "outputs": [],
   "source": []
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "8cd6362f",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.9.12"
  },
  "toc": {
   "base_numbering": 1,
   "nav_menu": {},
   "number_sections": true,
   "sideBar": true,
   "skip_h1_title": false,
   "title_cell": "Table of Contents",
   "title_sidebar": "Contents",
   "toc_cell": false,
   "toc_position": {
    "height": "calc(100% - 180px)",
    "left": "10px",
    "top": "150px",
    "width": "288px"
   },
   "toc_section_display": true,
   "toc_window_display": true
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
